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Motivated by recent NMR data suggesting the presence of a Fulde-Ferrell-Larkin-Ovchinnikov 
(FFLO) vortex lattice in magnetic fields perpendicular to the superconducting plane in the heavy- 
fermion superconductor CeCoIn5, we examine here the stability of FFLO states in quasi two- 
dimensional (2D) superconductors under a perpendicular field. It is found that a 3D-like portion 
of Fermi surfaces peculiar to CeCoIn5 stabilizes a FFLO vortex state modulating along the field 
parallel to the c-axis and that the direction of modulation in a FFLO vortex lattice is related to the 
characters of the normal to FFLO and the Abrikosov to FFLO transitions. 

PACS numbers: 



O ' I. INTRODUCTION 
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Recently, a series of observations**** in the heavy fermion superconductor CeCoIn5 at low temperatures and in high 
fields seem to have clarified that the new high field phase in the parallel field configuration (H || ab), surrounded by 
the discontinuous ff C 2-transition and a continuous transition from the Abrikosov vortex lattice, will be a Fulde-Ferrell- 
Larkin-Ovchinnikov (FFLO) vortex lattice. This FFLO stated is expected to have a one-dimensional modulation of 
the gap function along 5 H through the observation 2 of a reduction of ultrasound velocity in the new phase, which can 
be attributed to an additional vortex tilt assisted by an elastic fluctuation of the nodal planes of the FFLO structural. 
The presence of the nodal planes seems to be also consistent with the NMR datai A more recent NMR measurement^ 
in perpendicular fields (H || c), together with a subtle anomaly in heat capacity data—, has shown features similar to 
Q ■ the above-mentioned ones in H || ab. This similarity strongly suggests the presence of a FFLO vortex state with a 
modulation along the c-axis and the applied field. However, this fact may be unexpected, because the Fermi surface 
(FS) of CeCoIn5 is roughly cylindrical, and the FFLO modulation in the Pauli- limited case (i.e., with no vortices) 
' is not expected to develop along a direction with open FS. We point out here that the presence of a small sheet of 
, three-dimensional (3D) FS can support a FFLO modulation parallel to H, which is stable with respect to a vortex 
tilt. Further, the modulation direction of a possible FFLO state is found to be reflected in the characters of the H C 2 
, and FFLO transitions. 
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II. PHASE DIAGRAM AND FERMI SURFACE 

Our analysis is based on a microscopic derivation of a Ginzburg-Landau functional and is a straightforward 
extention— of that in Ref.5, where a 2nd order FFLO to Abrikosov transition field decreasing upon cooling was 
first demonstrated. Typical examples of phase diagrams we obtain at this time are shown in Fig.l where h denotes 
the magnetic field normalized by the orbital limiting field. Except for the used forms of FS, the material parameters in 
Fig.l (a) and (b) are the same as each other. Figure 1(a) follows from the use of a purely cylindrical FS, in which the 
in-plane component (Vp)ab of the Fermi velocity is much larger than (Vp) c everywhere, and, as in the Pauli-limited 
q I case, the FFLO modulation tends to be formed along the ab-plane everywhere in the phase diagram. Due to the 
Landau-level (LL) structure of the pair-field in the presence of vortices, however, a two-fold symmetric structure due 
. £h ! to the nodal planes parallel to H is not described in the lowest LL (n — 0) but rather in the next LL (n = 1). The 
nodal planes can be regarded as being accommodated as periodic stripes in the n = 1 vortex lattice— That is, a 
FFLO state modulating not along H but in a perpendicular direction to H may occur at low enough temperature ( 
t = T/T c q < 0.15 ) in Fig. 1(a). However, the normal to n=l vortex lattice transition, occurring on the thick dashed 
line below t < 0.15, is of 2nd order. On the other hand, in Fig. 1(b) where a cylindrical FS with small 3D-likc branches 
(the dashed lines in the Inset) was used, we have a 2nd order transition between the Abrikosov vortex lattice and 
a FFLO one modulating along H and with nodal planes perpendicular to H in much higher temperatures than the 
region dominated by the n = 1 LL. This FS in Fig. 1(b) simulates that expressed by the j3\ and /?2 bands^ of the FSs 
of CeCoIn5. As observed— in CeCoIn5 , the transition from the normal to this FFLO state is discontinuous 5 , while 
the FFLO to Abrikosov transition is of 2nd order. Notice, by comparing both the figures with each other, that the 
directions of modulation in FFLO states correlate with the characters of the transitions surrounding the FFLO state 
when the vortices are present. 
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FIG. 1: Examples of h v.s. t (= T/T c q) phase diagrams in H || c resulting (a) from the purely cylindrical FS and (b) from a 
cylindrical one with 3D-like portions (see the dotted lines in the Inset). Below To, the //^-transition occurs on each thick solid 
curve and is discontinuous, while each dotted curve is the 2nd order // C 2-transition curve and its extrapolation. The thin solid 
curve in (b) is the 2nd order transition line between the Abrikosov vortex lattice and the FFLO one with modulation || H. In 
both figures, the Maki parameter is 1.0. 



III. DISCUSSION 



Several remarks are needed to compare Fig. 1(b) with the experimental phase diagram 7 -. First, although the FFLO 
state with nodal planes || H inevitably appears at the lowest temperatures where the -//^-transition is of 2nd order, 
a small amount of impurities play the role of shifting various temperature scales to lower temperatures^ so that the 
n = 1 LL state is lost in real systems. Second, the experimental 2nd order Abrikosov to FFLO transition line is flat 
and insensitive 7 to t in contrast to that in Fig. 1(b). However, we find 6 - that a flat FFLO transition curve is obtained 
by slightly reducing the 3D portion of FS or by incorporating the antiferromagnetic (AF) fluctuation because an AF 
fluctuation, weaker with increasing h, competes with the Pauli paramagnetic effect. Further, we note that the FFLO 
state modulating along H in H || c is more stable than that in H || ab. In the latter, the nodal plane becomes softer 
with increasing h or the material anisotropy, and a reduction^ of tilt modulus occurs in the FFLO state, while the 
tilt modulus in the FFLO state in the former rather increases upon cooling 6 . Ultrasound experiments in H || c are 
useful for checking the validity of the present theory. 

The author thanks Y. Matsuda for informative discussions. 
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